Deduplicate recommendations

Learn how to remove variants from your recommendations by using Recommend deduplication.

With Recommend deduplication, you can refine your recommendations by removing item variants from your AI model training. The feature offers several advantages:

• Improves recommendation accuracy and speed by removing item variants before the training process.
• Improves recommendation quality by using events from all item variants.
• Separates deduplication from search so it doesn’t affect the search experience.

Recommend deduplication versus search deduplication

Algolia offers two types of deduplication:

• Search deduplication removes duplicate items from search results. This can help improve the relevance of the results and make them more user-friendly.
• Recommend deduplication remove duplicate items from Algolia AI recommendations. This can help improve the diversity of the recommendations and make them more personalized.

With Recommend deduplication, your AI models are trained with all variants, and the recommendations then filtered by the API before sending you the results to remove variants.

How Recommend deduplication works

Recommend deduplication adds two processes to your AI model training:

• Pre-training process: generates a training dataset with only one variant per item. It also merges all events from variants of the same item.
• Post-training process: add all the variants dropped during the pre-training process back into the final set of recommendations.

Item variants share the same recommendations.

Set up the deduplication for a model

To deduplicate your recommendations, you must first declare an attribute for distinguishing variants, then turn on deduplication when configuring a Recommend model. After that, verify that the recommendations have been deduplicated.

Configure an attribute for distinguishing variants

First, choose which attribute defines records as variants:

1. Select your Algolia application

   Go to the Algolia dashboard and select your Algolia application.

2. Select Search

   On the left sidebar, select Search.

3. Select your index

   Select your Algolia index:

   

1. Select deduplication

   On the *Configuration tab, go to the Deduplication and Grouping page.

2. Define variants

   In the Attribute for Distinct box, select or enter the attribute name you want to use to define variants.

   

   Only use the distinct option if you also want to deduplicate search results.

Enable Recommend deduplication on your model

1. Select your Algolia application

   Go to the Algolia dashboard and select your Algolia application.

2. Select Recommend

   On the left sidebar, select Recommend.

1. Select a model

   Create a new Recommend model or edit an existing one for the index you used when setting the attributeForDistinct.

2. Select deduplication

   In the section Distinct and deduplication of recommendations, select the Deduplicate recommendations option. The attribute you selected for defining variants is shown.

3. Continue configuration

   Continue to configure your Recommend model and click Save.

   

Verify the recommendations

To check that the deduplication is working for your recommendations, revisit the model configuration once the training process has finished:

1. Preview

   Go to the Preview section.

2. Check for duplicates

   Use the Search for a record box to search for an item that should have variants.

   This displays the list of recommendations for the selected item. They shouldn’t contain any variants.

Examples

The following examples illustrate how Recommend deduplication works. The index has records for T-shirts in different colors and sizes:

• One red T-shirt in one size (XS)
• Two green T-shirts in two sizes (S, M)
• Three blue T-shirts in three sizes (L, XL, XXL)

This example uses the Related Items model to recommend the top three similar items with and without deduplication (with the color attribute configured as attributeForDistinct).

Without deduplication

Without deduplication, recommendations include variants, such as blue (XXL) or green (M), except for the red T-shirt which doesn’t have any.

Base item	Recommendation 1	Recommendation 2	Recommendation 3
red (XS)	green (S)	blue (L)	blue (XL)
green (S)	green (M)	red (XS)	blue (L)
blue (L)	blue (XL)	blue (XXL)	green (M)

With deduplication

With deduplication, the recommendations don’t include any variants. However, since this example dataset only includes three records, it can only generate two recommendations. If you introduce a new item, such as an orange T-shirt, it will be added as a third recommendation.

Base item	Recommendation 1	Recommendation 2	Recommendation 3
red (XS)	green (S)	blue (L)	Doesn’t apply
green (S)	red (XS)	blue (L)	Doesn’t apply
blue (L)	red (XS)	green (M)	Doesn’t apply